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Key Takeaways 


• Kinect has evolved: 

• Whether you did it -> How you did it 

• One or two people -> A room of six people 

• Kinect opens up amazing opportunities for building 
richer and more engaging experiences 

• New generation of Kinect available later this month 
on Xbox One and next year on Windows 

• Get involved now: ID@Xbox and Kinect for 
Windows 
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Kinect is Evolving 
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What is NUI? 

Natural User Interaction (NUI) is a completely new way to 
experience technology. It is a paradigm shift that allows 
us to go beyond only manual input to communicating with 
our technology in a more natural way. 
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Multi-Modal 







KEYBOARD VISION 

+ 

MOUSE 


VOICE 


GESTURE 


TOUCH OTHER SENSORS 
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Playoke Dance 

Playoke is a dance game for professional fitness 
clubs -> players dance along to famous songs 

4 to 36 players can be play at the same time 

The Kinect takes a picture of the player and 
places the image on the screen next to others 

The dance video has been "tagged" so the 
Kinect can see the correct dance positions 

Movements of the all the players tracked in real 
time, analyzed & compared with others 
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Playoke System Overview 
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Kinect for Windows 1.8 SDK 


Kinect Fusion 

HTML5/JS 

Background 

Color 

App Model 

Removal API 

Color pixels added for 

Expanding access to 

"Green Screen" without 

new level of realism 

Kinect for web devs 

the actual screen 
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Kinect Fusion 
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Demo 

Kinect Fusion Color 


Photobombr 
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Kinect Common Bridge 

• Dead-simple Kinect integration into 3rd party libraries and 
graphics frameworks from MS Open Tech 

• Collab with Cinder & openFrameworks communities: native 
OF add-on and Cinder block available now 

• Apache 2.0 license 


W 


bW Microsoft 

g Open Technologies 


CJNDER 



http://aka.ms/KinectCommonBridqeBloq 
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Demo: Kinect Common Bridge 

Fireflies (openFrameworks) 


Particle Man (Cinder) 
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A Platform That Fully Understands 
the People in the Room 


Who they are 


• Identity 


What they are saying 


• Speech 


What they are doing 


• Face Tracking 

• Engagement 

• Skeieton tracking 

• Interactions 

• Hands states 

• Expression 

• Controiier Pairing 

• Video 



Saying^ 


Saying 


Who 

JohnA 


Engaged 

Dancing 

Focused 


Mappy 
Dancing 
Hands Ciosed 
Burning Caiorie 


ChrisC 


Cheering 


Not Engaged 
Hands 
Ciosed 
Seated , 
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NUI Platform 


Experiences 



r 




Xbox One 


Windows 



Hand State 


Expression 


HD Face 
Tracking 


Interactions 


Skeletal Tracking 


Identity 


Speech 


Video & 
Registration 


Controller 

Pairing 


NUI Service, Runtime & APIs 
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Sensor Capabilities 

• Wider field of view (horizontally and vertically) for depth and color 

• 3x depth fidelity 

• lOSOp, HD color camera (30fps) 

• Lighting independent. Active IR (30 fps) 

• Improved microphone array (zero balanced) 



Improved range of high quality operation .5M near, 4.5M far 
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Demo 

Augmented Reality 


NUI Evolution 
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Skeletal Tracking Features 


Improved reliability and accuracy 

• More reliable lock-on and more stable joints 

More anatomically correct skeleton 

• Hips in the right place, new shoulder parent 

Six players tracked at all times 

• Simplified engagement, bystander involvement 

Hand-tip and thumb joints 

• Enables subtle and more nuanced hand gestures 

Per-joint orientation 

• Great for character retargeting 



Lumbar 


Pelvic Girdle 


Femur 


Spinal Column 
Cervical 
Vertebrae 


Thoracic 
Vertebrae •" 


Clavicle 


Sternum 

Ribs 
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Hand Tips & Thumbs 


• High-precision 3D point 

• Hand tip has high precision, good for subtle 
motion 

• Analog open/close 

• Use hand length to get analog detail in 
between open/close hand states 

• Wrist flick gestures 

• Swiping, shooting, whipping, slapping, 
throwing 

• New interactions 



• Thumb in/out as button press, pinch gestures 
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Joint Orientation 


• Yaw, pitch, and roll for each joint 

• Generated primarily from skeletal-position data 

• Roll is smartly derived when possible (for 
example, arms use thumbs) 

• Exposed as quaternions in the body frame 

• Each is the absolute orientation of the parent 
bone 

• Basis for each joint is defined by: 

•Bone direction (Y) - always matches skeleton 
•Normal (Z) - joint roll, perpendicular to the bone 
. Binormal (X) - perpendicular to bone and normal 





GAME DEVELOPERS CONFERENCE NEXT® 


NOVEMBER 5-7, 2013' EXPO OATES: NOV 5-B LOS ANGELES, CA GOCNext.CDm 

L ■ 

Hand-State Features 

Real-tinne classification of open, closed, and lasso 

• Combine with motion for brand-new interactions! 

Improved quality 

• Much less "flicker" when palms are facing the sensor 

• Better with long sleeves, arms extended, hands rotated 

Visual Gesture Builder (VGB) integration 

• Add new states and improve reliability of hard poses 
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Xbox One Interaction Paradigm 


Common Build Menu Navigation 


• NEW APIs enables you to easily gesture enable your UI 

• For menuing and some gameplay 

• Consistent interaction with Xbox One System and other games 

• Benefit from Xbox One usability tests 

• Lowers your cost of developing basic interactions for your game menus 

• Navigation and manipulation: targeting, selection, press & scroll 


Custom Build Unique Game experience 


• Game experience and gestures : throw ball, golf swing, dance,... 

• Use updated Visual Gesture Builder to build unique and fun gestures 




GMfii^EVELOPERS CONFERENCE NE)CT® NOVEMBER 5-7, 2013'' LOS ANGELES, CA GOCNext.CDm 

\ % n 


Xbox One Interaction Language + 
System 


Engagement 

■ 

Targeting/ 

Pressing 

■ 

Scrolling/ 

Zooming 


Reusable Building Blocks 

cheaper to build menus/navigation, even with custom look 


Consistent User Experience 

your game and rest of Xbox One 
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Engagement 


Xbox 360 


Xbox One 

• Wave to engage 


• Look towards screen 



• Put hand in PhiZ 



• Open palm toward screen 


Improvements 




Smaller, but deliberate, barrier to interaction 
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Targeting & Pressing 


Xbox 360 

■ 


Xbox One 

• Hold and wait to press 

1 


• Press arm towards 
button, then release 

• As you progress with 



1 


pressing, cursor attracts 
towards button center 


Improvements 




More deliberate interaction 
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Scrolling & Zooming 


Xbox 360 


Xbox One 

• Swipe left/right 


• Grip hand over scrollable 

• No zooming 


area to start 


• Move left/right up/down 

• Release hand-grip to stop 



Improvements 


More user control over scrolling 
Not just page-left/page-right anymore 
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Demo 

Interactions 
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Building Custom Gestures 


Rinse and 
repeat 


VgbView 


Preview gestures on Xbox 
devkit 



• DiscreteGestureResults 

• ContinuousGestureResults 


NUIView [Record] 
XSConvert [Convert] 


Record example gestures using 
Xbox Studio and Convert 




Tag gestures using VGB 



Build and analyze 

gestures using VGB 
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Visual Gesture Builder 


K 


Visual Gesture Builder 


File View Help 


I Explorer 


SteeringWheel 
^ ^Steer.Left 

^^.jSteeringOO 
Steering© 1 
l^*Steering 02 
j^iSteeringOB 
[^^SteeringiM 
j^^SteeringOS 
|^*Steering 06 
|^ 4 Steering 07 
j^4Steering08 
|^« Steering 09 
[^^ SteeringlO 
Steering 11 
^*Steering 12 
I^SteeringlB 
^4Steering14 
^SteeringlS 
^>Steering 16 
^iSteeringl? 

I Steering 18 

|^«Steering 19 
|^ 4 Steering 23 
>teer_Lefta 
iSteer_Right 
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Movement : FramePrevious: LEFT FrameNext:RIOIT KeyframePrevious:Ctr1+LEFT KeyframeNext:Ctr1+RIGHT AttributePrevious:PAGEUP AttributeNext:PA6EDO*»J 
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Visual Gesture Builder 


K 


Visual Gesture Builder 


File View Help 


I Explorer 


|Steer_Lefta 
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Movement : FramePrevious: LEFT FrameNext:RIOIT KeyfrainePreviousrCtrl+LEFT KeyframeNext:Ctr1+RIWT AttributePrevious:PA6EUP AttributeNext:PA6EDO*(N 

Selection : SetCursorStart:Shift+HO«E SetCursorEnd: Shift+END FramePrevious: Shift+LEFT FrameNext:Shift+RIQHT KeyframePrevious:Ctr1+Shift+LEFT KeyframeNextiCtrl+Shift+RIGHT 
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Demo 

VGB View 


Steering Demo 
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Getting Involved 


ID@Xbox 

Independent Developers 
Publishing Program 
for Xbox One 



http://www.xbox.com/id 


KINECT V'/indows 


diTimuritty i l omg 



Announcing 1.8 SDK! Click here to learn more and download 
this release. 


http : //a ka . ms/k4wdev 
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Introducing : ID@Xbox 


• http://www.xbox.com/id 

• ID@Xbox == Independent Developers Publishing 
Program on Xbox One 

• Enables teams of all sizes to bring new and exciting 
content to Xbox One 

• Initial focus on developers in Xbox One regions -> 
longer term any retail Xbox One will be able to be used 
as a developer kit 
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Kinect for Windows 

http://aka.ms/k4wdev 

Download 1.8 SDK & toolkit now 

Build apps for Windows 7/8.1 (desktop) using C++, 
C#/VB, HTML5/JS 

Shipping pre-release v2 sensors to select developers 
this month (RTM in 2014) 
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In Closing 

• Kinect has evolved and opens up opportunities for 
building richer & more engaging experiences 

• New generation of Kinect available later this month on 
Xbox One and next year on Windows 

• Kinect Common Bridge 

http://aka.ms/KinectCommonBridae 

• Get involved now: 

• ID@Xbox http://www.xbox.com/id 

• Kinect for Windows http://aka.ms/K4WDev 


